Image forming apparatus, image forming method and computer-readable storage medium having an image forming program

ABSTRACT

An image forming process for easily malting an image drawing (ortho-image) in an actual place while confirming it in real time. Measuring of control points (at least three points) is performed (S 110 ), and preparation is made for forming an ortho-image based on image data thereof and survey data. Where photographing is carried out from a plurality of directions, additional image measuring is performed (S 140 ). Then, ortho-image formation is performed on a PC (S 160 ). Here, if a formed ortho-image is not what is desired, ortho-image correction is performed (S 180 ). After a satisfactory image is obtained (S 200 ), the process moves to a next area to be measured, and the same operation is repeated. On the other hand, unless a satisfactory image is obtained, additional image measuring (S 140 ) is performed again.

BACKGROUND OF THE INVENTION

[0001] The present invention relates to an image forming apparatus, animage forming method and a computer-readable storage medium having animage forming program. More particularly, the present invention relatesto a portable and simple image measuring technology utililized forsurveying and making a drawing in a land surveying field, which usesdigital processing by digital photogrammetry and a personal computer.

[0002] Specifically, the present invention relates to an image formingtechnology for forming a digital orthogonally projected image(ortho-image) from one to a plurality of images to be superposed on thedrawing when a drawing of a site to be measured is made. The presentinvention relates to an image forming technology which enables anyoperator to easily make a drawing of an orthogonally projected image byusing a survey instrument and images so as to stick an ortho-image tothe drawing, and also enables a digital orthogonally projected image tobe generated and rectified such that the situation of a site to bemeasured can be understood in detail. An object of the present inventionis to provide an image which enables stable stereo analysis to beperformed even for a stereo-image.

[0003] The present invention can also be applied to, for example, animage obtained by photographing a wide range divided into small imageshaving overlapping areas with one another.

[0004] According to a prior art, a drawing obtained by surveying in asite to be measured has been made by using paper and a pencil or thelike which are typically used in plane table surveying. In recent years,a line drawing of a site to be measured has been made by using a surveyinstrument and a portable computer in combination which are typicallyused in pen-based mapping system.

[0005]FIG. 18 is a view illustrating conventional stereo-imagemeasuring. Usually, as shown in the drawing, the three-dimensionalcoordinates are obtained by photographing two or more overlapped imagesbased on the principle of triangulation by stereo-image photographing.If a target range is wide, a plurality of images are photographed. Forexample, in even a simple site, photographing of ten to twenty images isusually necessary. In this case, unless orientation (i.e., calculating acamera position, inclination or the like) can be performed for eachstereo pair (model), a stereo-image cannot be rectified.Three-dimensional measuring cannot be performed, either.

SUMMARY OF THE INVENTION

[0006] In the conventional case, however, a drawing was expressed onlyby a line image even if it was made in the actual place, and it wasimpossible to sufficiently understand the situation of the site. Thus,the situation of the site was photographed by a camera. However, sincephotographing was performed from an oblique direction whereas adirection of a requested drawing was vertical, a photographed image hadno direct connection with the drawing. As a result, it was necessary foran operator to understand the situation of the site by comparing thedrawing with the image. This practice was inconvenient, andunderstanding of the site was difficult.

[0007] A range to be photographed with one image was limited. Even inthe case of photographing a plurality of images, because correlation andcontinuity among the respective photographs are lacking, this led to thedifficulty of obtaining consistency among the photographs, andcomparison with the drawing was complicated and difficult. If a highlyaccurate ortho-image was required, lens distortion of the camera was anobstacle. Conventionally, it was impossible to produce a highly accurateortho-image by a camera which has no data on lens distortion.

[0008] In the case of conventional land surveying, for surveying a siteto be photographed and making a drawing for the site, it was necessaryto survey an enormous number of points and perform three-dimensionalmeasuring. Thus, much man power and labor had to be expended. On theother hand, if a dose and terrestrial photogrammetry is used, indirectsurvey three-dimensional data can be obtained only by photographing. Butsince only a camera using a film was available conventionally,developing and scanning took time (e.g., two to three days), and severaldays were also expended for analyzing work. Unless processing up toanalysis was carried out after film development, there was no knowingwhether photographed image data was analyzable or not, and whetherstable, sure and highly reliable analysis was possible or not. As aresult, conventionally, this method has not been used so often becausethe necessity to again perform the work such as photographing mayhappen.

[0009] With popularized and increased use of digital cameras in recentyears, digital close and terrestrial photogrammetry can now be performedby a digital camera which is different from a conventional film oranalog camera. Use of this technology eliminates labor and time for filmdevelopment and scanning. Since analysis is performed by a computer, theprocess from photographing to analyzing can be finished within one totwo days.

[0010] However, after acquisition of photographic data, it was necessaryto bring the photographic data to the place of an analyzing device(computer) for analysis and measuring. Thus, photographing improper foranalysis might result in the necessity to again perform the work such asphotographing, and stable and sure analysis was not always carried out.

[0011] For performing analysis and measuring by images, if controlpoint/orienting point arrangement, a photographic range, photographicoverlapping or the like was not proper, stable and highly reliableanalysis was impossible. Consequently, unstable results occurred, andeven analysis itself was impossible.

[0012] The present invention was made with the foregoing points in mind.The object of the present invention is to provide an image formingapparatus for, when a drawing of the situation of a site to be measuredis made, easily making a drawing of the site from an orthogonallyprojected image (ortho-image) so as to facilitate understanding of thesituation. The object of the present invention is to quickly and easilygenerate/rectify an image in the site to be measured without any storageomission or mistakes, to understand the situation of the site on thespot, and to easily make an image drawing (ortho-image) in real timewhile confirming the same. The object of the present invention is toperform image complementary measuring only by simple photographing andsurveying of several points with a survey instrument, to simultaneouslyobtain an image drawing for facilitating understanding of the situationof the site, to interpolate portions not photographed or hard to makeout, and to produce an ortho-image of a high resolution and wide range.

[0013] Another object of the present invention is to facilitateunderstanding of a situation from a plurality of images, even if thesituation is hard to grasp, for example, some parts are difficult to seeor a far part becomes rough with one image, and produce a highlyaccurate ortho-image. The object of the present invention is tointegrate a plurality of images by repeating simple work and obtain awide-ranging ortho-image. Further, the object of the present inventionis to form a highly accurate and high-quality ortho-image at a highspeed by photographing a plurality of images overlapped with adjacentareas.

[0014] The still another object of the present invention is to correctlens distortion (camera calibration correction) by simple measuring, andsimultaneously form a highly accurate ortho-image even by a camerahaving no lens distortion data. Further, the object of the presentinvention is to form a good quality ortho-image of a necessary placewith required accuracy by using various survey instruments to rectifythe ortho-image while measuring the three-dimensional coordinates.

[0015] According to the present invention, the following operations andconfirmation can be performed in a site to be photographed:

[0016] (1) confirmation of a measuring range (e.g., overlapping);

[0017] (2) generation and confirmation of an ortho-image obtained byintegrating plurality of images;

[0018] (3) confirmation of the entire measuring/photographing range anda range to be drawn.

[0019] (4) confirmation of entire arrangement of controlpoints/orienting points; and

[0020] (5) confirmation of orientation, and confirmation of a modelformed for stereo method analysis.

[0021] In accordance with first solving means of the present invention,there is provided an image forming apparatus, which comprises:

[0022] a control point measuring section for measuring a centrallyprojected image having a plurality of control points, and obtaining theimage coordinates for said control points;

[0023] a coordinate transformation parameter calculating section forobtaining a transformation parameter for correlating, based on saidimage coordinates for said control points obtained by said control pointmeasuring section and the three-dimensional coordinates for actuallymeasured control points, said image coordinates with saidthree-dimensional coordinates;

[0024] an orthogonally projected image (ortho-image) forming section forforming an orthogonally projected image from said centrally projectedimage based on said transformation parameter obtained by said coordinatetransformation parameter calculating section; and

[0025] an ortho-image correcting section for correcting the imagecoordinates obtained by said ortho-image forming section based on thethree-dimensional coordinates for actually measured additional points,and then performing correction of said orthogonally projected image.

[0026] In accordance with second solving means of the present invention,there are provided an image forming method, an image forming apparatusand a computer-readable storage medium having an image forming program.In this case, the image forming method comprises:

[0027] an image input function, to which a plurality of images includingcontrol points or orienting points overlapped with one another areinputted;

[0028] a storage function for previously storing ground coordinatevalues of control points or orienting points;

[0029] an orthogonally projected image (ortho-image) forming functionfor forming an ortho-image from plurality of images inputted by saidimage input function based on image coordinate values, alternativelyphotographic coordinate value, and said ground coordinate values of thecontrol points or the orienting points; and

[0030] a determining function for determining, based on the ortho-imageformed by said ortho-image forming function, necessity ofre-photographing, and necessity of changing a photographing position,alternatively a control point or an orienting point position.

BRIEF DESCRIPTION OF THE DRAWINGS

[0031]FIG. 1 is a constitutional view of an image forming apparatus ofthe present invention.

[0032]FIG. 2 is a flow chart of an image forming process according to afirst embodiment of the present invention.

[0033]FIG. 3 is an explanatory view showing a case where a wide range ismeasured.

[0034]FIG. 4 is a flow chart of an image forming process according to asecond embodiment of the present invention.

[0035]FIG. 5 is a flow chart of on-line control point measuring.

[0036]FIG. 6 is a flow chart of automatic control point measuringperformed by a control point measuring section.

[0037]FIG. 7 is an explanatory view showing an example of control pointarrangement.

[0038]FIG. 8 is a flow chart of coordinate transformation parametercalculation process.

[0039]FIG. 9 is a flow chart of additional image measuring.

[0040]FIG. 10 is a flow chart of an ortho-image forming process.

[0041]FIG. 11 is a flow chart of ortho-image correction process.

[0042]FIG. 12 is a flow chart of displaying a control point shortagearea or an improper image area.

[0043]FIG. 13 is an explanatory view showing an example of a controlpoint shortage area or an improper image area.

[0044]FIG. 14 is a flow chart of displaying an image shortage place oran improper place.

[0045]FIG. 15 is an explanatory view showing displaying of an imageshortage place or an improper place.

[0046]FIG. 16 is an explanatory view showing the camera coordinates andthe model coordinates.

[0047]FIG. 17 is a flow chart of off-line processing.

[0048]FIG. 18 is an explanatory view showing conventional imagemeasuring and photographing.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0049] Now, the preferred embodiments of the present invention will bedescribed with datum to the accompanying drawings.

[0050] A. Constitution and Operation of Image Forming Apparatus

[0051]FIG. 1 is a constitutional view of an image forming apparatus ofthe present invention. The image forming apparatus comprises a controlunit 10, a storage section 2, an input/output interface 3, athree-dimensional coordinates input section 7, an image input section 4,a display section 5, and an input/output section 6. The control unit 10furthermore includes a control point measuring section 101, a coordinatetransformation parameter calculating section 102, an ortho-image formingsection 103, an ortho-image correcting section 104, and an additionalimage measuring section 105. These sections (functions) can be realizedby, for example, a portable computer (a PC).

[0052] The storage section 2 previously stores control point data, andstores image data or the like. The control point data is stored in, forexample, the three-dimensional coordinates (X, Y, Z) called groundcoordinates. The input/output interface 3 is composed of a common bus orthe like, and used for connecting various devices. The three-dimensionalcoordinates input section 7 obtains the three-dimensional coordinatesfor control points or additional measuring points by various surveyinstruments, e.g., a global positioning system (a GPS), a total station,and so on, or a three-dimensional coordinates measuring device. Theimage input section 4 obtains a two-dimensional or three-dimensionalimage from, for example, a digital camera. From the image input section4, one or a plurality of images including control points are inputted.

[0053] The display section 5 is provided with a CRT, a liquid crystaldisplay, a plasma display or the like, and performs two-dimensional orthree-dimensional displaying. The display section 5 displays anortho-image formed by the ortho-image forming section 103, theortho-image correcting section 104 of the control unit 10. The displaysection 5 also displays a control point position inputted by the controlpoint measuring section 101, an image added by the additional imagemeasuring section 105 or the like. The input/output section 6 transfersvarious image data and results of determination, alternatively thethree-dimensional coordinates data or the like from a three-dimensionalcoordinates input device such as a survey instrument or the like withother units. For the input/output section 6, various input/outputdevices, for example, an optical disk device, a card recording medium(e.g., an HDD, a memory or the like), a floppy disk, a keyboard, a pen,a mouse, a terminal and a CD-ROM disk drive, can be provided. Theinput/output section 6 gives instructions regarding control points andvarious positions on the screen of the display section 5 by a pointingdevice such as a mouse, a write pen or the like.

[0054] The control point measuring section 101 measures a centrallyprojected image having a plurality of imprinted control points, andobtains the image coordinates for the control points. The control pointmeasuring section 101 measures, by using, as the three-dimensionalcoordinate input section 7, a survey instrument for measuring a distanceand an angle from an already-known point, the three-dimensionalcoordinates for the control points based on the obtained coordinates,and stores the same in the storage section 2 as required. Depending onthe control points, the control point measuring section 101 canautomatically obtain each centrally projected image coordinate valuecorresponding to each control point for which the three-dimensionalcoordinates has been obtained.

[0055] The coordinate transformation parameter calculating section 102obtains a transformation parameter. This transformation parameter isused for transforming the image coordinates obtained by the controlpoint measuring section 101 into the photographic coordinates having anoriginal point set as a principal point (a point where a line passingthrough a lens center and orthogonal to the screen intersects theimage), and correlating, based on the image coordinates and thethree-dimensional coordinates for the control points, which have beenobtained by the control point measuring section 101, the imagecoordinates with the three-dimensional coordinates. Also, as describedlater, the coordinate transformation parameter calculating section 102can calculate, by satisfying a collinear condition for centralprojection and using an image input section where a principal distanceis known, a transformation parameter based on at least three controlpoints, for which a ground coordinate system is known. Further, thecoordinate transformation parameter calculating section 102 can correctlens distortion based on lens distortion data of the image input sectionor actually measured data of a plurality of control points.

[0056] The ortho-image forming section (ortho-image forming function)103 transforms the three-dimensional coordinates into the imagecoordinates based on the transformation parameter obtained by thecoordinate transformation parameter calculating section 102, and formsan orthogonally projected image. For forming an orthogonally projectedimage by sticking a plurality of images, the ortho-image forming section103 can select each image obtained based on a distance from each imageto a control point or a distance from each image to a measuring point.

[0057] Also, the ortho-image forming section 103 forms an ortho-imagefrom image coordinate values (or photographic coordinate values) of thecontrol points/orienting points and ground coordinate values based on aplurality of stereo images inputted by the image input section 4. Theortho-image forming section 103 can include a function of discriminatelydisplaying a non-overlapped area/non-photographed area determined by theortho-image correcting section 104 or a shortage/failed positioning ofthe control points/orienting points on the display screen of theortho-image. Further, the ortho-image forming section 103 can include afunction of executing orientation such as successive orientation,relative orientation or the like based on a plurality of stereo images.

[0058] The ortho-image correcting section 104 corrects the imagecoordinates obtained by the ortho-image forming section 103 based on thethree-dimensional coordinates for actually measured additional points,and performs correction of the orthogonally projected image. Also, theortho-image correcting section 104 can form, if a plurality of centrallyprojected images are obtained, an orthogonally projected image bycombining images selected according to a proper standard, e.g., imagesof places having specified/lower reduction scales or having relativelysmall reduction scales, alternatively images close to the measuringpoint or the control point, on priority basis. Further, the ortho-imagecorrecting section 104 can display on the display section 5, if aplurality of centrally projected images are obtained, an image shortagearea or an improper area which results from formation of theorthogonally projected image performed by the ortho-image formingsection 103.

[0059] A determining function included in the ortho-image correctingsection 104 determines, based on the ortho-image formed by theortho-image forming section 103, necessity of changing a photographingposition and necessity of changing positions of the controlpoints/orienting points. This determining function includes a functionof extracting a non-overlapped area, the area being not overlapped by atleast two stereo images in the ortho-image. A determining section 12includes a function of extracting a non-photographed area which is notcovered with by stereo images in a measuring target area. Also, thedetermining function includes a function of determining ashortage/failed positioning of control points/orienting points includedin a portion overlapped by at least two stereo images in theortho-image. Further, the determining function includes a function ofoutputting, if results of orientation performed by the ortho-imageforming section 11 are not converged, an instruction for changingsetting of the control points/orienting points to both or either of theortho-image forming section 103 and the additional image measuringsection 105.

[0060] A correction data forming function included in the ortho-imagecorrecting section 104 forms data for re-photographing if thedetermining function detects a non-overlapped area/non-photographed areaor a shortage/failed positioning of the control points/orienting pointsin the ortho-image. This data for re-photographing is necessary forsolving the detected failure, and for this data, for example, dataregarding a photographing position, data regarding a photographingrange, data regarding selection of control points/orienting points, datafor specifying control points/orienting points to be rearranged or thelike can be used. Such data is fed back to the ortho-image formingsection 103.

[0061] The additional image measuring section 105 measures othercentrally projected images so as to include the control points measuredby the control point measuring section 101, and calculatestransformation parameters for these centrally projected images by usingthe coordinate transformation parameter calculating section 102.

[0062] Next, an overall operation will be summarized.

[0063]FIG. 2 is a flow chart of an image forming process according to afirst embodiment of the present invention. The drawing shows a casewhere an image is formed based on one or more images andthree-dimensional coordinate values by a survey instrument. An imageforming process carried out in the control unit 10 is describedhereinbelow by referring to FIG. 2.

[0064] The process shown in FIG. 2 has two courses of processing on-lineprocessing performed in an actual place, and off-line processing forperforming only control point measuring in the actual place and bringingback remaining jobs to be finished in an office or the like. In theon-line processing, different from the case of the off-line processing,control point measuring, subsequent image forming, displaying, and soon, are carried out in the actual place.

[0065] In the image forming process, first measuring of control points(three points or more) is performed (step S110), and preparation is madefor forming an ortho-image based on image data thereof and survey data.Furthermore, if photographing is carried out from a plurality ofdirections, additional image measuring is performed (step S140). Forperforming analysis only with one image, the additional image measuring(step S140) is unnecessary. Then, for example, on a PC, ortho-imageformation is performed (step S160). Here, if a formed ortho-image is notwhat is desired, ortho-image correction is performed (step S180). Inthis ortho-image correction work, determination is made as to whether asatisfactory image is obtained or not (step S200). If “OK”, then theprocess moves to a next area to be measured, and the same operation isrepeated from the control point measuring (step S110). On the otherhand, if no satisfactory image is obtained in the ortho-image correctionwork (step S180), then the additional image measuring (step S140) iscarried out again, and the foregoing operations are repeated until asatisfactory image is obtained.

[0066] The additional image measuring (step S140) is performed, forexample if some parts are hard to see when photographing is carried outfrom one direction, if an image of a far part becomes rough even on oneimage and thus sufficient accuracy is not provided, or the like. Inother words, if an ortho-image is formed by performing furtherphotographing from another direction for compensating for an alreadyinputted image or yet another direction where there is a shortage, aplace hard to see or a place of less accuracy will beselected/synthesized from a camera image of another direction, andthereby a good quality ortho-image will be formed.

[0067]FIG. 3 is a view illustrating a case where a wide range ismeasured. If a range to be measured is wide, a measuring range may bedecided in accordance with necessary accuracy, and the image formingprocess in FIG. 2 may be repeated sequentially in respective areas ofmeasuring ranges 1, 2, 3, . . . and so on, of FIG. 3. Necessary accuracydepends on, for example, a distance from an object, a size of one CCDpixel or the like. Thus, by dividing a wide range for processing, anortho-image can be easily formed no matter how wide a range is.

[0068]FIG. 4 is a flow chart of an image forming process according to asecond embodiment of the present invention. The process shown is imagemeasuring processing for performing stable measuring by stereo images.Image measuring processing is described hereinbelow by referring to FIG.4.

[0069] First, control points/orienting points are set, and measured datais set by a survey instrument (step S1100). The control points have beenaccurately measured on the basis of a ground coordinate system (absolutecoordinate system) (X, Y, Z) beforehand by the survey instrument. If theorienting points are arranged uniformly (discretely) on a photographicimage, it is not necessary to measure positions thereof beforehand.Needless to say, the orienting points can be also used asposition-measured control points. The ground coordinate system (X, Y, Z)is a three-dimensional orthogonal coordinate system where real space ofa model has been set.

[0070] Then, photographing is performed such that each image cansufficiently overlap another, and the control points/orienting points,for example at least six points or more, can overlap one another (stepS1200). Here, as an example, each image has at least six or more controlpoints/orienting points, however, the proper number thereof can be usedas occasion demands. For example, if bundle adjustment or the like isused, the number of control points for measuring can be reduced tothree. Even if orienting points are used instead of control points, thenecessary number of targets is about six. Alternatively, four points areenough if orienting points can be assumed that they are arranged on aplane.

[0071] After the photographing, various set values necessary for imageprocessing, e.g., both or either of an ortho-image forming range andresolution, and so on, are set onto the portable computer (a PC) (stepS1300). Then, the control points/orienting points of each image aremeasured on a PC (step S1400). Here, by the ortho-image formingfunction, orientation and ortho-image formation are automaticallycarried out according to a program (step S1500). The specific processingwill be described in detail later.

[0072] After the calculation by the ortho-image forming function, theresults of orientation and an ortho-image are displayed on the displayscreen (step S1600). Then, based on the results of step S1600, eachcheck item is confirmed for the measured image (step S1700).

[0073] The display function displays arrangement of all and each imageby the ortho-image. Furthermore, the insufficient results of orientationare discriminately displayed based on various bits of displayinformation by which the check items are confirmed. Regarding the checkitems, if the results of orientation are insufficient (e.g., targettedaccuracy is not achieved, no convergence occurs or the like), improperselection or arrangement of the control points/orienting points isdetermined. A failure of an entire measuring range such as existence ofa non-photographed area caused by omission of photographing or the likeis confirmed by the formed ortho-image. If a non-overlapped area existsbecause of an image overlapping failure, propriety of an overlappedportion is confirmed. If there is such an overlapping failure, byextracting only an overlapped part between adjacent images in the formedortho-image and displaying the same, a gap is formed in a joined partbetween images on the ortho-image, and thus immediate determination canbe made.

[0074] In addition to the confirmation of the display screen by ameasuring operator, the check items can be automatically confirmed bythe control function (especially, the determining function). If theresults of orientation do not converge as in the case of failedarrangement or failed positioning of the control points, or ifsuccessive orientation is not normally carried out, the results oforientation containing errors are outputted in the step S1600 isoutputted, and a failure such as an improper image is discriminatelydisplayed. For example, the control function can allow the displayfunction to automatically display the fact that the results oforientation fail to achieve the targetted accuracy or fail to convergeby an identifier, a flag, and can also allow the display function todiscriminately display a proper range by color or patterns. Even in thecase where a non-photographed area or a non-overlapped area exists, thearea can be properly recognized by the control function and using animage processing technology. Specifically, for example, a vacant placeof an image is detected, and by identifying an area/position thereof,the above failures can be determined.

[0075] If necessity of correction is decided based on the results ofchecking, then a decision is made as to necessity of determining,because of a shortage, a positioning failure of the controlpoints/orienting points, arrangement thereof again for re-photographing(step S1720).

[0076] Here, for example, if rearrangement of the controlpoints/orienting points is not necessary because only an imagephotographing position is not proper, as in the case where anon-photographed area or a non-overlapped area exists, data forre-photographing is formed by the correction data forming function, andphotographing is performed again according to this data (step S1860).Then, the process is resumed again from the step S1400.

[0077] On the other hand, for example, if the arrangement of the controlpoints/orienting points must be changed as in the case where the resultsof orientation are not converged or successive orientation is notcarried out normally, rearrangement thereof is examined (step S1740). Ifthere are other control points/orienting points on the image and theimage coordinates for these points is obtained without re-photographing,selection of the control points/orienting points is changed from thealready photographed image (step S1760), and the above calculation isstarted from the step S1500. This processing may be performedautomatically. On the other hand, if other proper controlpoints/orienting points do not exist on the image, arrangement changingis performed by moving the arrangement of the control points/orientingpoints (step S1840). Then, data for re-photographing the image includingthe rearranged control points/orienting points is formed by thecorrection data forming function, and re-photographing is performedaccording to the formed data (step S1860). Then, the process is executedfrom the step S1400.

[0078] The re-photographing processing in the step S1860, the measuringprocessing in the step S1400 may be executed only for the imageincluding the control points/orienting points, which is necessary tohandle each failure, alternatively for all the images to bephotographed.

[0079] The decision as to necessity of re-photographing based on theresults of checking, and the decision as to the step to which theprocess returns when re-photographing is necessary, may be manually madeby the operator based on the display screen, alternatively madeautomatically by a PC, otherwise made by proper combination of themanual operation and a PC.

[0080] As described above, when it is decided that any correction isnecessary correction is determined based on the result of checking inthe step S1700, the process is finished with the decision that a desiredortho-image has been obtained.

[0081] B. Detail of the Processing

[0082] Next, the flow chart of FIG. 2 will be described in detail.

[0083] (1) Control Point Measuring Processing by the Control PointMeasuring Section 101 (Step S110).

[0084] Hereinbelow, control point measuring processing (step S110) bythe control point measuring section 101 is described in detail.

[0085] (1-1) On-Line Measuring

[0086] First, an example of on-line measuring will be described. FIG. 5is a flow chart of on-line control point measuring. In on-line controlpoint measuring for forming an ortho-image on the field of a work site,work is done while forming and confirming an ortho-image in an actualplace. Thus, according to the on-line control point measuring,non-efficient work such as re-measuring the control points in the actualplace can be avoided, such work being necessary in the case of measuringmistakes or shortage parts found after a photographed image is broughtback and analyzed.

[0087] First, control points, three points at the minimum, are set in arange to be measured (step S112). Furthermore, the control points arepreferably be discriminable targets on an image and measured by a surveyinstrument (three-dimensional coordinate input section 7). For thecontrol points, for example, prisms or reflecting targets can be used,alternatively those tartgets which return for returning reflecting(measuring) signals can be used in the case of a non-prism totalstation. Then, photographing is performed by a digital camera (imageinput section 4) (step S114). This photographing may be performed forone image, or the desired number of images from a plurality ofdirections. For example, when some parts are hard to see if photographedfrom one direction, when image accuracy is considered, it is better toperform further photographing from a plurality of directions.

[0088] Then, the image photographed by the digital camera and stored inthe storage device of the camera is transferred to the control unit 10of the image forming apparatus and inputted to the storage section 2(step S116). The image inputted to the control unit 10 comes to bemeasured on a PC, and the photographed image is displayed on the displaysection 5. Here, the targets are collimated by the survey instrument(step S118) and confirmed. If OK, the control points as targets aremeasured on the image on a PC (step S120). In this case, for example,positions of the control points are instructed by a pen as the inputdevice of the input/output section 6. After the image coordinates (px,py) (CCD coordinate system, and an original point is set to be a screenend such as the left upper end of the screen) is measured, a measuringinstruction is outputted to the three-dimensional coordinates inputsection 7, measuring is performed by the three-dimensional coordinatesinput section 7 (step S122), and then measured ground coordinate valuesof the control points are sent through the input/output I/F 3 to thecontrol unit 10 and the storage section 2 (step S124). Accordingly,correlation of the ground coordinates for the control points with theimage coordinates on a PC and measuring are completed. Then, the samework is repeated for the minimum three points (step S126).

[0089] Next, automatic control point measuring by the control pointmeasuring section 101 will be described.

[0090]FIG. 6 is a flow chart of automatic control point measuringperformed by the control point measuring section. FIG. 7 is a viewshowing an example of arrangement of control points.

[0091] First, control points having reflecting sheets are set (stepS501). Attaching the reflecting sheets facilitates automatic measuringwith an image and simultaneous measuring work by a total station, anon-prism total station. The control points are set as targets to beseen from any directions, and these points can be fixed to, for example,apexes of triangle pyramids or triangle poles. If the number of controlpoints is three, for example, arrangement like that shown in the drawingis set, and it is advisable to set one obviously different in size.Then, as one example, photographing is carried out with a strobe light(step S503). Accordingly, any situations can be surely detected. Then,the photographed image is inputted to the personal computer (step S505).Then, the three-dimensional coordinates are measured by the surveyinstrument (step S507). For measuring the control point having thelargest reflecting sheet, information thereof is transferred as anattribute is simultaneously transferred from the survey instrument tothe control unit 10.

[0092] Then, by template matching (image correlation processing), theimage coordinates on the image is detected (step S509). For example, alight quantity returns from each of the reflecting sheets has highluminance, and by registering the shapes of the control points astemplates (matching patterns) beforehand, position detection isfacilitated. For the templates, in this example, two kinds of small andlarge templates can be prepared. Then, the image coordinates for theposition-detected control points are correlated with points of thethree-dimensional coordinates (step S511). For example, by arrangingthree points and setting one among them to be large as shown in thedrawing, the large control point on the image becomes discriminable bythe template, and is correlated based on the attribute measured on thethree-dimensional coordinates. The other two points can be correlatedbased on the three-dimensional coordinates positions and the imagecoordinates positions thereof even if these points are photographed fromany directions.

[0093] For increasing reliability of position detection, two images arephotographed, one with a strobe light and the other without a strobelight, and by taking a difference between these two images, only thetargets emerge. Thus, automatic coordinate position detection bymatching becomes surer and more reliable. Accordingly, detection can besurely carried out from any directions.

[0094] The processing has been described with datum to the large andsmall reflecting sheets. Image coordinates detection and correlating canalso be performed by coloring or patterning the reflecting sheets. Otherthan the reflecting sheets, any discriminal materials on the image canbe used. Accordingly, there are no limitations placed on the number,arrangement, shapes, and so on of control point.

[0095] Next, template matching processing will be described.

[0096] For template matching, any proper method such as a normalizedcross correlation method, a sequential similarity detection algorithm(SSDA) or the like may be used. For example, by using the SSDA, aprocessing speed can be increased. Hereinbelow, the SSDA is described.

[0097] An expression of the SSDA is shown below. In this expression, apoint where a residual R(a,b) becomes smallest is an image position tobe obtained. For increasing a processing speed, for example, in additionof this expression, the addition is finished when a value of R(a,b)exceeds a minimum value of a past residual, and calculation is performedso as to move to next (a,b). $\begin{matrix}{{R\left( {a,b} \right)} = {\sum\limits_{{m1} = 0}^{{N1} - 1}{\sum\limits_{{n1} = 0}^{{N1} - 1}{{{I_{({a,b})}\left( {m_{1},n_{1}} \right)} - {T\left( {m_{1},n_{1}} \right)}}}}}} & (1)\end{matrix}$

[0098] T(m₁, n₁): Template image,

[0099] I_((a,b))(m₁, n₁): Partial image of target image

[0100] (a,b): Left upper coordinates of template image

[0101] R(a,b): Residual

[0102] (1-2) Coordinate Transformation Parameter Calculation Processingby the Coordinate Transformation Parameter Calculating Section 102 (StepS300)

[0103] If the foregoing measuring for the control points is OK, thencoordinate transformation parameter calculation by the coordinatetransformation parameter calculating section 102 is performed (stepS300).

[0104]FIG. 8 is a flow chart of coordinate transformation parametercalculation process.

[0105] First, since the points measured on a PC are obtained as theimage coordinates on a charge coupled device (a CCD) such as a digitalcamera, these coordinates are transformed into a photographic coordinatesystem (x, y) (step S203). The photographic coordinate system (x, y) isthe two-dimensional coordinates having an original point set as aprincipal point. On the other hand, the image coordinates are CCDcoordinate system (px, py), for example, the two-dimensional coordinateshaving an original point set in the left upper side.

[0106] The three points measured on the ground coordinate system (X, Y,Z) by measuring of the survey instrument and the two points of the imagecoordinate system (x, y) measured on a PC are substituted forexpressions below, and each parameter for coordinate transformation(coordinate transformation parameter) is obtained (step S205).Expressions 2 and 3 are collinear condition expressions where aprojection center, an image on a CCD and an object are located on astraight line. Accordingly, if there are at least three already-knownpoints, each coordinate transformation parameter can be calculated. Inthis case, however, a principal distance c must be nearly known. If acalibrated camera is used, no problem occurs since a principal distancec is known. $\begin{matrix}{x = {{- c}\frac{{a_{11}\left( {X - X_{0}} \right)} + {a_{12}\left( {Y - Y_{0}} \right)} + {a_{13}\left( {Z - Z_{0}} \right)}}{{a_{31}\left( {X - X_{0}} \right)} + {a_{32}\left( {Y - Y_{0}} \right)} + {a_{33}\left( {Z - Z_{0}} \right)}}}} & (2) \\{y = {{- c}\frac{{a_{21}\left( {X - X_{0}} \right)} + {a_{22}\left( {Y - Y_{0}} \right)} + {a_{23}\left( {Z - Z_{0}} \right)}}{{a_{31}\left( {X - X_{0}} \right)} + {a_{32}\left( {Y - Y_{0}} \right)} + {a_{33}\left( {Z - Z_{0}} \right)}}}} & (3)\end{matrix}$

[0107] Herein,

[0108] x, y: Image coordinates,

[0109] x₀,y₀: Principal point position,

[0110] X, Y, Z: Ground coordinates,

[0111] X₀, Y₀, Z₀: Coordinates of projection center,

[0112] c: Principal distance

[0113] a_(ij): Rotation matrix

[0114] ω, φ, κ: Inclination of camera

[0115] On the other hand, if a principal distance c is not known, thisvalue can be obtained by measuring coordinate values of four points on aplane based on a two-dimensional projective transformation. Also, byusing six points, a relationship between the image coordinates and thethree-dimensional coordinates (target points coordinates) of the objectcan be obtained by an approximate solution based on a three-dimensionalprojective transformation expression (direct linear transformation).

[0116] Hereinbelow, a method by the three-dimensional transformationexpression is described. This processing is for calculating atransformation parameter for obtaining a pixel position on anortho-image. Here, the processing is performed mainly based on knowncoordinates for the control points/orienting points and the measuredimage (photographic) coordinates.

[0117] First, a basic expression is shown below. $\begin{matrix}{{x = \frac{{L_{1}X} + {L_{2}Y} + {L_{3}Z} + L_{4}}{{L_{9}X} + {L_{10}Y} + {L_{11}Z} + 1}}{y = \frac{{L_{5}X} + {L_{6}Y} + {L_{7}Z} + L_{8}}{{L_{9}X} + {L_{10}Y} + {L_{11}Z} + 1}}} & (4)\end{matrix}$

[0118] Herein, (x, y) Image coordinates,

[0119] (X, Y, Z): Ground coordinates,

[0120] L₁ to L₁₁: Coordinates transformation parameters

[0121] By calculating the expression 4 by using a least squares methodbased on data of the control points, coordinate transformationparameters L₁ to L₁₁ for deciding a relationship between the imagecoordinates (x, y) and the ground coordinates (X, Y, Z) can be obtained.

[0122] Further, a coordinate transformation expression is not limited tothe foregoing, and any proper coordinate transformation expressions canbe employed as long as these can correlate the ground coordinates withthe image coordinates.

[0123] (1-3) Camera Calibration

[0124] The foregoing description has been made of the case wheremeasuring is performed with accuracy for permitting lens distortion tobe ignored or a non-distortion lens is used. However, if there is lensdistortion and this distortion cannot be ignored for accuracy, measuringis carried out with a camera having pre-calculated lens distortion orwhile correcting lens distortion by processing described below.

[0125] Specifically, in the case of using a camera having no lensdistortion data, the lens distortion is corrected by measuring six ormore control points and obtaining lens distortion of the camera based onthe following expressions 5 to 7. In the case of a camera having precalculated lens distortion, coordinate values are obtained whilecorrecting the lens distortion by using these expressions from thestart. $\begin{matrix}{x = {{{- c}\frac{{a_{11}\left( {X - X_{0}} \right)} + {a_{12}\left( {Y - Y_{0}} \right)} + {a_{13}\left( {Z - Z_{0}} \right)}}{{a_{31}\left( {X - X_{0}} \right)} + {a_{32}\left( {Y - Y_{0}} \right)} + {a_{33}\left( {Z - Z_{0}} \right)}}} + {\Delta \quad x}}} & (5) \\{y = {{{- c}\frac{{a_{21}\left( {X - X_{0}} \right)} + {a_{22}\left( {Y - Y_{0}} \right)} + {a_{23}\left( {Z - Z_{0}} \right)}}{{a_{31}\left( {X - X_{0}} \right)} + {a_{32}\left( {Y - Y_{0}} \right)} + {a_{33}\left( {Z - Z_{0}} \right)}}} + {\Delta \quad y}}} & (6)\end{matrix}$

 Δx=x ₀ +x(k ₁ r ² +k ₂ r ⁴)

Δy=y ₀ +y(k ₁ r ² +k ₂ r ⁴)

r ²=(x ² +y ²)/c ²  (7)

[0126] k₁, k₂: parameters of radical distortion

[0127] These expressions are calculated by measuring six or more controlpoints on the ground coordinates and the image coordinates and by asuccessive approximation solution.

[0128] In addition to the foregoing, by using a direct lineartransformation with self-calibration, correction of lens distortion canbe performed by expressions 8 and 9. In this case, however, at leasteight control points are necessary. $\begin{matrix}{{x + {\Delta \quad x}} = \frac{{L_{1}X} + {L_{2}Y} + {L_{3}Z} + L_{4}}{{L_{9}X} + {L_{10}Y} + {L_{11}Z} + 1}} & (8) \\{{y + {\Delta \quad y}} = \frac{{L_{5}X} + {L_{6}Y} + {L_{7}Z} + L_{8}}{{L_{9}X} + {L_{10}Y} + {L_{11}Z} + 1}} & (9)\end{matrix}$

[0129] By canceling denominators of the above expressions based on dataof the control points and by using the least squares method, unknownvariables k₁, k₂, x₀ and y₀ can be obtained.

[0130] As described above, lens distortion is obtained by the foregoingexpressions, and accordingly, measuring while correcting lens distortioncan be performed. These expressions are only examples, and calculationmay be carried out by other expressions.

[0131] Then, referring again to FIG. 5, after the foregoing coordinatetransformation parameters calculation processing (step S300) isfinished, by using the transformation parameters, a residual between thecoordinate transformation values and the control points is obtained, anda decision is made as to whether the residual is within a prescribedvalue or not (step S128). If the residual is within the prescribedvalue, then the process moves to the next processing. On the other hand,if the residual is not within the prescribed value, the process returnsto the step S118, and while increasing the number of points to bemeasured, the operations of the steps S118 to S128 are performed untilthe residual comes within the prescribed value.

[0132] Here, the residual can be obtained in the following manner.Specifically, the transformation parameters obtained by the expressions2 and 3 (alternatively, the expressions 5 and 6) and the imagecoordinates (x, y) measured for the control points are substituted inexpressions 10 and 11 described below, and calculation control points X′and Y′ of the ground coordinate system (X, Y, Z) are calculated. Then, aresidual with the actually measured ground coordinate values (X, Y) areobtained by an expression 12. If the residual δ thus obtained is withinthe prescribed value, it is OK For the prescribed value, for example,accuracy necessary in an actual place is set. Herein, n means the numberof control points. Expressions for the residual should not be limited tothe ones described here, but any other expressions can be used.$\begin{matrix}{X^{\prime} = {{\left( {Z - Z_{0}} \right)\frac{{a_{11}x} + {a_{21}y} - {a_{31}c}}{{a_{13}x} + {a_{23}y} - {a_{33}c}}} + X_{0}}} & (10) \\{Y^{\prime} = {{\left( {Z - Z_{0}} \right)\frac{{a_{12}x} + {a_{22}y} - {a_{32}c}}{{a_{13}x} + {a_{23}y} - {a_{33}c}}} + Y_{0}}} & (11)\end{matrix}$

 δ={square root}{square root over (Σ(X′−X)²+Σ(Y′−Y)²)}/n  (12)

[0133] (2) Additional Image Measuring Processing by the Additional ImageMeasuring Section 105 (Step S140)

[0134] Next, additional image measuring processing (step S140) will bedescribed in detail. FIG. 9 is a flow chart of additional measuring. Ifthe number of images to be photographed is one, alternatively if animage is formed with one image temporarily, this processing can beskipped. Also, if a plurality of images are photographed first in stepsS114 and S116, a step of photographing (step S142) and a step ofinputting an image to a PC (step S144) can be skipped.

[0135] Photographing of additional images is performed so as to includeall the control points set by the control point setting (step S112)carried out in the control point measuring processing (step S110). Then,the images are inputted to a PC (step S144), the additionallyphotographed images are displayed on the display section 5, and anadditional image to be measured is selected (step S146). Then, thecontrol points imprinted on the selected image are measured on a PC(step S148). This work is repeated by a number of times equal to thenumber of control points (step S150). For example, if a portablecomputer of a pen input type is used, by indicating the control pointsas targets by a pointing device such as a pen, the image coordinates(px, py) (or the photographic coordinates (x, y)) thereof is obtained.Then, coordinate transformation parameter calculation like thatdescribed above is performed (step S300).

[0136] Further, for additionally measuring other images, the processreturns to the additional image section (step S146), and this processingis repeated by a number of times equal to the number of images to bephotographed. When no additional measuring is carried out, the processmoves to next step to form an ortho-image (step S160).

[0137] (3) Ortho-Image Formation Processing by the Ortho-Image FormingSection 103 (Step S160)

[0138] Next, ortho-image formation processing (step S160) will bedescribed in detail. FIG. 10 is a flow chart of an image formingprocessing.

[0139] First, a ground coordinate for each pixel on an ortho-image iscalculated (step S207). In this processing, for forming an ortho-image,the image coordinates (x, y) for the ortho-image are transformed intothe ground coordinates (X, Y, Z). The ground coordinates (X, Y, Z) iscalculated by using the transformation parameters previously obtained inthe step S205 of the coordinate transformation parameter calculationprocessing (step S300). In other words, the ground coordinates (X, Y, Z)corresponding to the image coordinates (x, y) for the ortho-image isprovided by the following expression. Accordingly, a position forobtaining each pixel on the ortho-image can be obtained. $\begin{matrix}{{X = {X_{0} + {x\quad \Delta \quad X}}}{Y = {Y_{0} - {y\quad \Delta \quad Y}}}{Z = {- \frac{{aX} + {bY} + d}{c}}}} & (13)\end{matrix}$

[0140] Herein,

[0141] (X₀, Y₀): Position of left upper side of ortho-image in groundcoordinate system,

[0142] (ΔX, ΔY): Size of one pixel in ground coordinate system (e.g.,m/pixel),

[0143] (x, y): Image coordinates for ortho-image

[0144] (X, Y, Z): Ground coordinates,

[0145] a, b, c, d: Coefficients of plane equation formed by a pluralityof control points interpolating certain image coordinates (x, y).

[0146] This time, by using the transformation parameters obtained instep S205, the image coordinates (x, y) corresponding to the groundcoordinates (X, Y, Z) obtained in step S207 is calculated based on theexpressions 2 and 3 or the expressions 5 and 6 (step S209). From theobtained image coordinates (x, y), a gray value on the groundcoordinates (X, Y, Z) for the proper image is obtained. This gray valueis a pixel gray level in a two-dimensional position (X, Y) on theortho-image. In this way, a gray value of an image to be stuck to aposition (X, Y) on the ground coordinates is obtained. By performing theforegoing processing for all the pixels of the ortho-image, imagesticking is carried out (step S211).

[0147] Hereinbelow, sticking of a plurality of images is described more.

[0148] If the number of images is not singular but plural, thenselection of an image is decided based on a positional relationshipbetween the control points and the camera. In other words, a position(X₀, Y₀, Z₀) of a camera for each image is calculated based on theexpressions 2 and 3. Accordingly, a distance is obtained from valuesthereof and actually measured values (X, Y, Z) of control pointcoordinates, a pixel gray value of a closer image is obtained and thenthe image is stuck. Thus, by carrying out image sticking by using animage closer to the control point, image data having high resolution canbe selected.

[0149] (4) Ortho-Image Correction Processing by the Ortho-ImageCorrecting Section 104 (Step S180)

[0150] Next, ortho-image correction (step S180) will be described indetail. FIG. 11 is a flow chart of ortho-image correction.

[0151] Ortho-image formation (step S160) is performed, checking is madeon the formed ortho-image visually or by later described processing. Ifthere are measuring mistaken parts or parts which especially needmeasuring data, measuring is performed for these parts the surveyinstrument (step S184). The three-dimensional coordinates (groundcoordinates) measured is transferred from the survey instrument to thecontrol unit 10 (step S188), the above-described ortho-image formation(step S160) is automatically executed, immediate displaying of theortho-image is carried out by the display section 5, and then checkingis made again.

[0152] Accordingly, since measuring is performed point by point andprocessing is performed while confirming a corrected image in real time,measuring mistakes or measuring omissions can be prevented, and thusmeasuring can be always performed while confirming an image as an endproduct.

[0153] Next, a flow illustrating a method of extracting and correcting acontrol point shortage place or an improper place from the formedortho-image will be described.

[0154]FIG. 12 shows a flow chart of control point shortage place orimproper image place displaying. FIG. 13 is an explanatory view showingan example of a control point shortage area or an improper image area.

[0155] First, control point shortage place displaying is executed. Theortho-image formed from one or a plurality of images is first displayed(step S601). Then, a range in a measuring area which is intended to beconfirmed is specified (step S603). Here, whether control points arelocated in a proper control points arranging area or not is checked, anddisplayed (step S605). For example, if the number of control points issix, a display is like that shown in FIG. 13. For checking control pointarrangement, a proper control point range is divided on the ortho-imagebased on the number of control points to be measured, the coordinates ofthe control points are checked, decision is made as to whether thecontrol points are located in the divided sections or not, and a resultof the decision is displayed. Alternatively, a frame of a proper controlpoint range like that shown in FIG. 13 (in this example, the area isdivided into six sections) is displayed. But no limitation is placed ona method for deciding a proper control point range.

[0156] Subsequently, displaying of an improper image place is executed.First, a gap and a difference of elevation between adjacent controlpoints are calculated (step S607). Here, a point of which a calculatedvalue (e.g., a difference of elevation) is equal to or higher than athreshold value is displayed so as to show the fact (step S609). Thethreshold value is calculated from an image reduction scale such that adrastically changed point can be detected from, for example, the numberof control points or a gap and a difference of elevation between thecontrol points.

[0157] Then, as described above, in ortho-image correction (step S180),a control point shortage part and a part in the vicinity of the point ofwhich the calculation value is equal to or higher than the thresholdvalue are measured by the survey instrument, and then the image iscorrected. If no satisfactory result is obtained even by theseprocedures, it is advisable to execute additional image measuring basedon the information regarding these displayed images.

[0158] By the above measuring and processing, the control point shortagepart or the improper part of the ortho-image can be corrected.

[0159] Next, a method for extracting the shortage part or the improperpart from the formed ortho-image and displaying the same will bedescribed. Hereinbelow, a method of specifying a shortage/improper partbased on image resolution will be described.

[0160]FIG. 14 shows a flow chart of displaying an image shortagepart/improper part. FIG. 15 is an explanatory view showing displaying ofan image shortage part/improper part. Necessary measuring accuracy isset according to an object as it differs from object to object.

[0161] First, the ortho-image formed from one or a plurality of imagesis displayed (step S701). Here, cases of performing photographing frompositions A and B are assumed. Then, a range in a measuring area whichis intended to be confirmed is specified (step S703). Since a cameraposition (X₀, Y₀, Z₀) has been obtained based on the expressions 2 and3, accuracy of one pixel in the measuring confirming area is calculatedfrom this position (step S705). If photographing is performed fromplural directions, the above calculation is executed for each camera.Then, a place where pixel accuracy of each camera does not achieve setaccuracy and a range for overlapping are displayed on the ortho-image(step S707). Then, additional image measuring is performed from thedirection of the displayed place (step S709). Herein, a case ofphotographing additional images from an additional image photographingposition C is described.

[0162] Accordingly, an image for compensating for the image resolution(one pixel accuracy) of the image shortage place/improper place isobtained and, as a result, a satisfactory image can be obtained. Thepixel accuracy can be determined, for example, in a manner that accuracyis reduced as a ratio of transforming a centrally projected image intoan ortho-image is increased, or as a distance from an imagephotographing point to an object having an imprinted image is increased.

[0163] (5) Orientation

[0164] Next, orientation of each model will be described in detail. Theorientation is, for example, a part of processing in step S1500 shown inFIG. 4.

[0165] Based on the three-dimensional coordinates of each image obtainedin step S1400 shown in FIG. 4, orientation can be sequentially performedfrom the measuring points (e.g., six or more measuring points) such asthe control points/orienting points imprinted in each model image. Theorientation is data used for checking in step S1700.

[0166] First, as an example, relative orientation is described. Forstereo images, positions of left and right cameras are respectivelyobtained by calculation shown below.

[0167]FIG. 16 is an explanatory view showing the camera coordinates andthe model coordinates.

[0168] In the camera coordinate system, a lens center (projectioncenter, principal center) is set as an original point, and x and y axesare in parallel with x and y axes of the photographic coordinates. If aprincipal distance is c (c>0), then a point on a photographic plane isexpressed by (x, y, −c). The model coordinate system (X, Y, Z) is athree-dimensional coordinate system for defining a stereo image formedfrom a pair of two stereo photographs, and an original point thereof isset in the projection center of the left photograph.

[0169] First, parameters are calculated under a coplanarity conditionexpression like the following. $\begin{matrix}{{\begin{matrix}X_{01} & Y_{01} & Z_{01} & 1 \\X_{02} & Y_{02} & Z_{02} & 1 \\X_{1} & Y_{1} & Z_{1} & 1 \\X_{2} & Y_{2} & Z_{2} & 1\end{matrix}} = 0} & (14)\end{matrix}$

[0170] Herein,

[0171] X₀₁, Y₀₁, Z₀₁: Projection center coordinates of left imageexpressed by model coordinate system,

[0172] X₀₂, Y₀₂, Z₀₂: Projection center coordinates of right imageexpressed by model coordinate system,

[0173] X₁, Y₁, Z₁: Image coordinates of left image expressed by modelcoordinate system,

[0174] X₂, Y₂, Z₂: Image coordinates of right image expressed by modelcoordinate system.

[0175] Now, as shown in the drawing, an original point of the modelcoordinate system is set in the projection center of the left side, anda line connecting the projection center of the right side is set in theX axis. For a scale, a base line length is set to a unit length.Parameters obtained at this time are five rotational angles, i.e., arotational angle κ₁ of the Z axis and a rotational angle φ₁ of the Yaxis of the left camera, and a rotational angle κ₂ of the Z axis, arotational angle φ₂ of the Y axis and a rotational angle ω₂ of the Xaxis of the right camera. In this case, it is not necessary to considera rotational angle ω₁ of the X axis of the left camera as it is zero.

[0176] Under the above condition, a coplanarity condition expression 15becomes one like that shown below, and each parameter can be obtained bycalculating this expression. $\begin{matrix}{{F\left( {\kappa_{1},\varphi_{1},\kappa_{2},\varphi_{2},\omega_{2}} \right)} = {{\begin{matrix}Y_{1} & Z_{1} \\Y_{2} & Z_{2}\end{matrix}} = {{{Y_{1}Z_{2}} - {Y_{2}Z_{1}}} = 0}}} & (15)\end{matrix}$

[0177] Herein, between the model coordinates (X, Y, Z) and the cameracoordinates (x, y, z), a relational expression of coordinatetransformation like that shown below is established. $\begin{matrix}{{\begin{pmatrix}X_{1} \\Y_{1} \\Z_{1}\end{pmatrix} = {\begin{pmatrix}{\cos \quad \varphi} & 0 & {\sin \quad \varphi} \\0 & 1 & 0 \\{{- \sin}\quad \varphi_{1}} & 0 & {\cos \quad \varphi_{1}}\end{pmatrix}\begin{pmatrix}{\cos \quad \kappa_{1}} & {{- \sin}\quad \kappa_{1}} & 0 \\{\sin \quad \kappa_{1}} & {\cos \quad \kappa_{1}} & 0 \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}x_{1} \\y_{1} \\{- c}\end{pmatrix}}}{\begin{pmatrix}X_{2} \\Y_{2} \\Z_{2}\end{pmatrix} = {{\begin{pmatrix}1 & 0 & 0 \\0 & {\cos \quad \omega_{2}} & {{- \sin}\quad \omega_{2}} \\0 & {\sin \quad \omega_{2}} & {\cos \quad \omega_{2}}\end{pmatrix}\begin{pmatrix}{\cos \quad \varphi_{2}} & 0 & {\sin \quad \varphi_{2}} \\0 & 1 & 0 \\{{- \sin}\quad \varphi_{2}} & 0 & {\cos \quad \varphi_{2}}\end{pmatrix}\begin{pmatrix}{\cos \quad \kappa_{2}} & {{- \sin}\quad \kappa_{2}} & 0 \\{\sin \quad \kappa_{2}} & {\cos \quad \kappa_{2}} & 0 \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}x_{2} \\y_{2} \\{- c}\end{pmatrix}} + \begin{pmatrix}1 \\0 \\0\end{pmatrix}}}} & (16)\end{matrix}$

[0178] By using the foregoing expressions, unknown parameters areobtained based on the following procedure.

[0179] 1. An initial approximate value is usually set to 0.

[0180] 2. A value of differential coefficient is obtained by theexpression 7 when the coplanarity condition expression (expression 13)is tailor-developed around the approximate value and made linearization,and then an observation equation is established.

[0181] 3. By using the least squares method, a correction amount isobtained for the approximate value.

[0182] 4. The approximate value is corrected.

[0183] 5. By using the corrected approximate value, operations from 2 to5 are repeated until results converge.

[0184] No convergence may occur, for example, as in the case wherearrangement of the control points, orienting points is improper. In sucha case, in the displaying of orientation results in step S1600 of FIG.4, an improper image is discriminated, and then displayed.

[0185] If the results of calculating unknown parameters converge,another step, for example, successive orienting, is performed. Thesuccessive orientation is for setting an identical coordinate system byunifying inclinations or reduction scales of the respective modelssubjected to orientation and inter-connecting a plurality of images. Inthe successive orientation, a pair of stereophotographs are connected toeach other by fixing one orientation parameter and operating only theother orientation parameter. If this processing is carried out, aconnection range represented by the following expression is calculated.

ΔX _(j)=(X _(jr) −X _(jl))/(Z ₀ −Z _(jl))

ΔY _(j)=(Y _(jr) −Y _(jl))/(Z ₀ −Z _(jl))

ΔZ _(j)=(Z _(jr) −Z _(jl))/(Z ₀ −Z _(jl))

ΔD _(j)={square root}{square root over ((ΔX _(j) ² +ΔY _(j) ²))}  (17)

[0186] Herein,

[0187] (ΔX_(jl)Δ_(jl)YΔZ_(jl)): j-th left model of course coordinatesystem

[0188] (ΔX_(jr)ΔY_(jr)ΔZ_(jr)): j-th right model of course coordinatesystem

[0189] If ΔZ_(j) and ΔD_(j) are equal to a predetermined value (e.g.,0.0005 ({fraction (1/2000)})) or lower as a result of calculation, it isdecided that the successive orientation is normally executed.

[0190] If results of calculation do not converge, or if normal executionof the successive orientation is not decided, then an error is outputtedin the orientation result display in the step S1600 of FIG. 4, and afailure, such as an improper image or the like, is discriminablydisplayed.

[0191] C. Application

[0192] (1) Use of Various Survey Instruments

[0193] Next, use of various survey instruments and advantages will bedescribed.

[0194] For example, if correction work is carried out by the non-prismtotal station, in the case of an object which returns a light (measuringsignal), an ortho-images is renewed as occasion demands only byperforming measuring for the object. Accordingly, use of such a surveyinstruments is extremely effective for facilitating work and providingcapability of measuring a number of points.

[0195] Use of another survey instruments is very effective in that ifcorrection processing is carried out by an auto-tracking total station,only by walking with a prism in a shortage measuring area, anortho-image can be formed with higher accuracy.

[0196] Use of the other survey instruments is also greatly effective inthat if an area can receive date from satellites, only by walking with aGPS in the area, an ortho-image is renewed, and any operator can form anortho-image while easily performing complementary measuring.

[0197] For obtaining a high-density and highly accurate ortho-imagewithout any failures, it is advisable to perform stereo photographing tooverlap ranges to be measured, and perform relative orientation anactual place. In this way, automatic stereo matching (image correlationprocessing) is performed from stereo images to obtain high-densitythree-dimensional measuring points and, by forming an image by theortho-image forming section, high-density and highly accurateortho-image can be obtained. Alternatively, a shortage part, adiscontinuous part or the like is manually measured on the display whileseeing formed stereo images, and by combining the measuring pointsthereof and then forming an ortho-image by the ortho-image formingsection, an ortho-image having high luminance can be obtained withoutany failure.

[0198] (2) Measuring of Wide-Ranging Area

[0199] A wider range can be measured by executing, after the measuringfor the measuring range 1 shown in FIG. 3 is over, the process againfrom START of the image formation shown in FIG. 2. In this case, nospecial consideration is necessary for a boundary between the ranges tobe measured. Alternatively, by intentionally overlapping adjacentranges, e.g., the measuring ranges 1 and 2 with each other, and placingsome control points in the area thereof, control point measuring ofthese points by the survey instrument can be omitted, and it is onlynecessary to measure images of these points (step S148). Also, byincreasing an overlapping rate, images can be stereo-measured by usingthe second embodiment of the present invention. Accordingly, by usingthe images and control points of the measuring ranges, 1, 2, so as toform an ortho-image in each area, an ortho-image unifying the measuringranges 1, 2, . . . can be easily formed. Thus, gradual expansion of areato be measured can be facilitated.

[0200] (3) Off-Line Measuring

[0201] The control point measuring (step S110) was an example forperforming on-line measuring of control points. Next, off-lineprocessing will be described. FIG. 17 shows a flow chart of off-lineprocessing.

[0202] In the case of off-line processing, only photographing andcontrol point measuring by the survey instrument are performed in anactual place, and all other operations are done off-line (e.g., in anoffice or the like). Off-line measuring in control point measuring (stepS110) is divided into, for example, image photographing and controlpoint measuring, and these operations are carried out separately in ajob site. Results of these operations are then used for performingthorough analysis in the office or the like. In this case, a pluralityof images are obtained beforehand, and image formation is performedlater while selecting a proper image.

[0203] Accordingly, for the control point measuring, the surveyinstrument and a PC are not associated with each other (in other words,a PC is unnecessary), and measuring is carried out only with the surveyinstrument (step S132). Then, control point data obtained by themeasuring are batch-transferred to a PC (step S136). Then, additionalimage measuring (step S140) is performed. Thus, in the case of off-lineprocessing, since control point measuring and photographing are carriedout separately, an image obtained by aerial photographing performed by ahelicopter, a balloon or the like can be processed.

[0204] An image measuring program for executing the foregoing imagemeasuring method of the present invention can be provided by a recordingmedium such as a CD-ROM, a floppy disk or the like.

What is claimed is:
 1. An image forming method comprising: an imageinput function, to which stereo images including control points ororienting points overlapped with one another are inputted; a storagefunction for previously storing ground coordinate values of controlpoints or orienting points; and a determining function for determining,based on the stereo images formed by said image input function,necessity of re-photographing and necessity of changing a photographingposition, a control point or an orienting point position; wherein anortho-image is formed from stereo images inputted by said image inputfunction based on image coordinate values or photographic coordinatevalues and said ground coordinate values of the control points or theorienting points.
 2. An image forming method according to claim 1,further comprising: a display function for displaying the ortho-image.3. An image forming method according to claim 1, wherein said imageinput function includes a function of inputting stereo images as aplurality of images, and said determining function includes a functionof extracting a non-overlapped area, the area being not overlapped bythe stereo images in said ortho-image.
 4. An image forming methodaccording to claim 1, wherein said determining function includes afunction of extracting a non-photographed area in a measuring targetarea, which is not covered with images in said ortho-image.
 5. An imageforming method according to claim 1, further comprising: an ortho-imageforming function including a function of discriminatably displaying anon-overlapped area or a non-photographed area determined by saiddetermining function, or a shortage or a failed positioning of thecontrol points or the orienting points on a display screen of saidortho-image.
 6. An image forming method according to claim 1, whereinsaid determining function further includes a correction data formingfunction of forming data for re-photographing, where a non-overlappedarea or a non-photographed area, or a shortage or a failed positioningof the control points or the orienting points, is detected in saidortho-image.
 7. An image forming method according to claim 1, furthercomprising: an ortho-image forming function including a function ofexecuting orientation based on a plurality of images, and wherein saiddetermining function includes a function of outputting, if results ofsaid orientation do not converge, an instruction to change setting ofthe control points or the orienting points to said ortho-image formingfunction.
 8. An image forming method according to claim 7, wherein saidfunction of executing orientation is successive orientation.
 9. An imageforming method according to claim 7, wherein said function of executingorientation is relative orientation.
 10. An image forming apparatuscomprising: means for achieving an image forming method comprising: animage input function, to which stereo images including control points ororienting points overlapped with one another are inputted; a storagefunction for previously storing ground coordinate values of controlpoints or orienting points; and a determining function for determining,based on the stereo images formed by said image input function,necessity of re-photographing and necessity of changing a photographingposition, a control point or an orienting point position; wherein anortho-image is formed from stereo images inputted by said image inputfunction based on image coordinate values or photographic coordinatevalues and said ground coordinate values of the control points or theorienting points.
 11. An image forming apparatus according to claim 10,wherein said method further comprises: a display function for displayingthe ortho-image.
 12. An image forming apparatus according to claim 10,wherein said image input function includes a function of inputtingstereo images as a plurality of images, and said determining functionincludes a function of extracting a non-overlapped area, the area beingnot overlapped by the stereo images in said ortho-image.
 13. An imageforming apparatus according to claim 10, wherein said determiningfunction includes a function of extracting a non-photographed area in ameasuring target area, which is not covered with images in saidortho-image.
 14. An image forming apparatus according to claim 10, saidmethod further comprising: an ortho-image forming function including afunction of discriminatably displaying a non-overlapped area or anon-photographed area determined by said determining function, or ashortage or a failed positioning of the control points or the orientingpoints on a display screen of said ortho-image.
 15. An image formingapparatus according to claim 10, wherein said determining functionfurther includes a correction data forming function of forming data forre-photographing, where a non-overlapped area or a non-photographedarea, or a shortage or a failed positioning of the control points or theorienting points, is detected in said ortho-image.
 16. An image formingapparatus according to claim 10, said method further comprising: anortho-image forming function including a function of executingorientation based on a plurality of images, and wherein said determiningfunction includes a function of outputting, if results of saidorientation do not converge, an instruction to change setting of thecontrol points or the orienting points to said ortho-image formingfunction.
 17. An image forming apparatus according to claim 16, whereinsaid function of executing orientation is successive orientation.
 18. Animage forming apparatus according to claim 16, wherein said function ofexecuting orientation is relative orientation.
 19. A computer-readablestorage medium having an image forming program for achieving an imageforming method, said image forming method comprising: an image inputfunction, to which stereo images including control points or orientingpoints overlapped with one another are inputted; a storage function forpreviously storing ground coordinate values of control points ororienting points; and a determining function for determining, based onthe stereo images formed by said image input function, necessity ofre-photographing and necessity of changing a photographing position, acontrol point or an orienting point position; wherein an ortho-image isformed from stereo images inputted by said image input function based onimage coordinate values or photographic coordinate values and saidground coordinate values of the control points or the orienting points.20. A computer-readable storage medium according to claim 19, whereinsaid method further comprises: a display function for displaying theortho-image.
 21. A computer-readable storage medium according to claim19, wherein said image input function includes a function of inputtingstereo images as a plurality of images, and said determining functionincludes a function of extracting a non-overlapped area, the area beingnot overlapped by the stereo images in said ortho-image.
 22. Acomputer-readable storage medium according to claim 19, wherein saiddetermining function includes a function of extracting anon-photographed area in a measuring target area, which is not coveredwith images in said ortho-image.
 23. A computer-readable storage mediumaccording to claim 19, wherein said method further comprises: anortho-image forming function including a function of discriminatablydisplaying a non-overlapped area or a non-photographed area determinedby said determining function, or a shortage or a failed positioning ofthe control points or the orienting points on a display screen of saidortho-image.
 24. A computer-readable storage medium according to claim19, wherein said determining function further includes a correction dataforming function of forming data for re-photographing, where anon-overlapped area or a non-photographed area, or a shortage or afailed positioning of the control points or the orienting points, isdetected in said ortho-image.
 25. A computer-readable storage mediumaccording to claim 19, wherein said method further comprises: anortho-image forming function including a function of executingorientation based on a plurality of images, and wherein said determiningfunction includes a function of outputting, if results of saidorientation do not converge, an instruction to change setting of thecontrol points or the orienting points to said ortho-image formingfunction.
 26. A computer-readable storage medium according to claim 25,wherein said function of executing orientation is successiveorientation.
 27. A computer-readable storage medium according to claim25, wherein said function of executing orientation is relativeorientation.
 28. An image forming apparatus adapted for carrying out animage forming method, said image forming method comprising: an imageinput function, to which stereo images including control points ororienting points overlapped with one another are inputted; a storagefunction for previously storing ground coordinate values of controlpoints or orienting points; and a determining function for determining,based on the stereo images formed by said image input function,necessity of re-photographing and necessity of changing a photographingposition, a control point or an orienting point position; wherein anortho-image is formed from stereo images inputted by said image inputfunction based on image coordinate values or photographic coordinatevalues and said ground coordinate values of the control points or theorienting points.
 29. An image forming apparatus according to claim 28,wherein said method further comprises: a display function for displayingthe ortho-image.
 30. An image forming apparatus according to claim 28,wherein said image input function includes a function of inputtingstereo images as a plurality of images, and said determining functionincludes a function of extracting a non-overlapped area, the area beingnot overlapped by the stereo images in said ortho-image.
 31. An imageforming apparatus according to claim 28, wherein said determiningfunction includes a function of extracting a non-photographed area in ameasuring target area, which is not covered with images in saidortho-image.
 32. An image forming apparatus according to claim 28, saidmethod further comprising: an ortho-image forming function including afunction of discriminatably displaying a non-overlapped area or anon-photographed area determined by said determining function, or ashortage or a failed positioning of the control points or the orientingpoints on a display screen of said ortho-image.
 33. An image formingapparatus according to claim 28, wherein said determining functionfurther includes a correction data forming function of forming data forre-photographing, where a non-overlapped area or a non-photographedarea, or a shortage or a failed positioning of the control points or theorienting points, is detected in said ortho-image.
 34. An image formingapparatus according to claim 28, said method further comprising: anortho-image forming function including a function of executingorientation based on a plurality of images, and wherein said determiningfunction includes a function of outputting, if results of saidorientation do not converge, an instruction to change setting of thecontrol points or the orienting points to said ortho-image formingfunction.
 35. An image forming apparatus according to claim 34, whereinsaid function of executing orientation is successive orientation.
 36. Animage forming apparatus according to claim 34, wherein said function ofexecuting orientation is relative orientation.
 37. An image formingapparatus for image formation, said image forming apparatus comprising:an image input section, to which stereo images including control pointsor orienting points overlapped with one another are inputted; a storagesection to store ground coordinate values of control points or orientingpoints; and a determining section adapted to determine, based on thestereo images formed by said image input section, necessity ofre-photographing and necessity of changing a photographing position, acontrol point or an orienting point position; wherein an ortho-image canbe formed from stereo images inputted by said image input section basedon image coordinate values or photographic coordinate values and saidground coordinate values of the control points or the orienting points.38. An image forming method comprising: inputting stereo imagesincluding control points or orienting points overlapped with one anotherto an image input apparatus; storing ground coordinate values of controlpoints or orienting points in a storage apparatus; and determining,based on the inputted stereo images, the necessity of re-photographingand the necessity of changing a photographing position, a control pointor an orienting point position; wherein an ortho-image is formed fromstereo images inputted by said inputting based on image coordinatevalues or photographic coordinate values and said ground coordinatevalues of the control points or the orienting points.